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PREFACE 


This  Note  describes  issues  that  must  be  addressed  if  the  contribution  of 
intelligence  and  electronic  warfare/target  acquisition  (IEW/TA)  systems  to  combat 
outcomes  is  to  be  reliably  assessed  through  the  use  of  simulation  models.  The  analysis 
reported  here  was  done  in  the  context  of  the  project  "Measuring  the  Operational  Value  of 
Intelligence,  Electronic  Warfare  and  Target  Acquisition  (OPVIEW),"  which  was 
undertaken  by  the  Arroyo  Center  for  the  Army  under  RAND’s  Applied  Technology 
Program.  Its  objective  is  to  develop  a  methodological  approach  and  supporting  model  as 
tools  for  analyzing  and  measuring  the  operational  value  of  IEW/TA  in  combat  outcome 
terms.  This  Note  documents  general  considerations  that  motivated  the  approach  being 
taken  on  the  OPVIEW  project  The  design  of  the  OPVIEW  modeling  environment  and 
associated  models  will  be  described  in  subsequent  documents. 

The  Arroyo  Center  is  the  U.S.  Army’s  federally  funded  research  and  development 
center  (FFRDQ  for  studies  and  analysis  operated  by  RAND.  The  Arroyo  Center 
provides  the  Army  with  objective,  independent  analytic  research  on  major  policy  and 
management  concerns,  emphasizing  mid-  to  long-term  problems.  Its  research  is  carried 
out  in  five  programs:  Policy  and  Strategy;  Force  Development  and  Employment; 
Readiness  and  Sustainability;  Manpower,  Training,  and  Performance;  and  Applied 
Technology. 

Army  Regulation  5-21  contains  basic  policy  for  the  conduct  of  the  Arroyo  Center. 
The  Army  provides  continuing  guidance  and  oversight  through  the  Arroyo  Center  Policy 
Committee,  which  is  co-chaired  by  the  Vice  Chief  of  Staff  and  by  the  Assistant  for 
Research,  Development,  and  Acquisition.  Arroyo  Center  work  is  performed  under 
contract  MDA903-86-C-0059. 

The  Arroyo  Center  is  housed  in  RAND’s  Army  Research  Division.  RAND  is  a 
private,  nonprofit  institution  that  conducts  analytic  research  on  a  wide  range  of  public 
policy  matters  affecting  the  nation’s  security  and  welfare. 

Lynn  E.  Davis  is  Vice  President  for  the  Army  Research  Division  and  Director  of 
the  Arroyo  Center.  Those  interested  in  further  information  concerning  the  Arroyo  Center 
should  contact  her  office  directly: 
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Lynn  E.  Davis 
RAND 

1700  Main  Street 
P.O.  Box  2138 

Santa  Monica,  CA  90407-2138 
Telephone:  (213)  393-0411 
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SUMMARY 


The  Operational  Value  of  intelligence  and  electronic  warfare  (OPVIEW)  project 
is  endeavoring  to  develop  means  for  evaluating  intelligence  and  electronic  warfare/target 
acquisition  (IEW/TA)  systems  in  terms  of  their  contribution  to  combat  outcomes  through 
the  use  of  simulation.  The  challenge  of  adequately  representing  the  intelligence  process 
in  a  simulation  model  is  being  met  through  model  representations.  These  will  be 
described  in  subsequent  documents.  A  greater  and  more  fundamental  challenge  is  to 
contend  with  potentially  large  sensitivities  of  combat  outcomes  to  multiple  assumptions 
having  unresolvablc  uncertainties. 

Many  details  of  future  combat  situations  are  quite  uncertain,  which  poses  a 
problem  for  the  use  of  simulation  in  assessing  combat  outcomes.  Furthermore,  the 
inclusion  of  IEW/TA  causes  a  substantial  increase  in  the  amount  of  uncertainty  above 
what  must  be  addressed  in  simulations  directed  primarily  toward  lethal  systems. 
Consequently,  the  use  of  simulation  in  evaluating  the  contribution  of  IEW/TA  to  combat 
outcomes  requires  innovative  approaches. 

The  problem  of  uncertainties  in  models  is  generally  met  through  sensitivity 
analysis  where  possible  errors  in  inputs  to  the  model  are  estimated  and  corresponding 
error  estimates  on  the  outputs  are  calculated.  Unfortunately,  for  most  realistic  simulation 
models  of  complex  processes  the  number  of  runs  per  case  required  for  a  thorough 
sensitivity  analysis  is  astronomical.  If  simulation  models  are  to  make  a  contribution  to 
evaluating  IEW/TA,  the  number  of  excursions  required  to  bound  output  uncertainties 
must  be  kept  tractable. 

Aggressive  sensitivity  analysis  can  be  pursued  in  several  ways: 

•  Aggregated  modeling  limits  both  the  number  of  uncertainties  explicitly 
considered  and  the  time  for  individual  runs. 

•  Transparent  modeling  allows  for  rapid  model  revision,  facilitating  the  use  of 
multiple  model  variants  for  exploration  of  the  effects  of  nonparametric 
assumptions. 

•  Question-driven  model  development  provides  focused  modeling  limiting 
irrelevant  factors. 
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•  Selectively  varying  the  resolution  of  the  models  allows  for  structuring  the 
search  of  the  universe  of  cases  and  optimizing  the  utilization  of  analytic 
resources. 

•  Analytic  strategies  provide  a  "top-down"  structuring  of  the  list  of  cases  that 
must  be  ran. 

For  BEW/TA  systems,  a  large  portion  of  the  value  derived  from  situation 
assessment  comes  from  both  avoiding  being  surprised  and  exploiting  special 
opportunities.  Consequently,  the  incremental  value  of  an  IEW/TA  system  will  manifest 
itself  primarily  in  special  cases  where  such  surprises  or  opportunities  exist.  Where  the 
added  value  of  an  IEW/TA  system  is  being  estimated  through  a  series  of  simulated  cases 
sampling  plausible  situations,  such  critical  cases  should  be  oversampled.  A  generic  study 
might  involve  discovering  "interesting"  cases  through  an  exploration  of  the  space  of 
plausible  models  and  then  studying  the  implications  of  different  IEW/TA  attributes  on 
those  cases.  The  process  could  well  be  iterative  as  new  cases  are  identified  in  the  course 
of  analysis. 

A  specially  designed  computer  environment  could  support  this  style  of  modeling 
IEW/TA's  effect  on  combat  outcomes  in  several  ways,  particularly  through 

•  Programming  languages  and  software  tools  supporting  interactive  and 
adaptive  modeling. 

•  Aggregated  and  parameterized  baseline  models  that  may  be  modified  in  the 
context  of  specific  studies. 

•  Transparent  model  computer  code. 

These  facilities  would  permit  running  many  cases  in  support  of  any  specific 
analysis  of  combat  value.  Generating  these  runs  may  be  regarded  as  the  first  half  of  an 
analysis  of  combat  value.  The  second  half  is  to  generate  a  coherent  picture  of  all  the 
relevant  tradeoffs,  to  boil  down  the  complexities  of  the  sensitivity  analysis  to  a 
representation  that  will  provide  useful  input  to  decisionmaking. 

Such  a  simulation  environment  would  be  useful  for  asking  not  only  "What  is  the 
value  of  an  IEW/TA  system"  but  also  "When  (under  what  conditions)  will  it  have  that 
value?"  and  "Why  does  it  have  value?"  This  tool  could  prove  useful  for  examining 
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alternative  mixes  of  systems,  understanding  synergistic  effects  (e.g.  cueing),  and  seeing 
implications  for  employment  issues.  Consequently,  while  OPVIEW  has  focused  on 
measuring  the  combat  value  of  IEW/TA,  the  product  should  provide  not  just 
measurements  but  rather  decision  support  for  a  broad  range  of  issues.  OPVIEW  and 
other  projects  should  endeavor  to  produce  a  decision  aid  that  will  illuminate  both 
tradeoffs  and  essential  uncertainties.  Such  a  tool  could  be  of  value  to  decisionmakers 
who  must  consider  costs  and  risks  as  well  as  benefits. 
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I.  CHALLENGES  TO  ASSESSING  THE  VALUE  OF  IEW/TA  BY 
SIMULATING  COMBAT 


The  relative  value  of  systems  for  intelligence  and  electronic  warfare/target 
acquisition  (IEW/TA)  may  be  determined  in  a  variety  of  ways  including  the  comparison 
of  technical  characteristics,  ability  to  provide  estimations  of  commanders’  information 
need,  or  subjective  judgment  by  experts.  The  OPVIEW  project  is  endeavoring  to 
develop  means  for  evaluating  IEW/TA  systems  in  terms  of  their  contribution  to  combat 
outcomes.  Such  an  evaluation  would  permit  comparisons  between  diverse  systems  and 
force  components  and  evaluation  of  the  combined  value  of  groups  of  systems. 

Simulation  coula  oe  a  powerful  analytic  tool  for  determining  the  contribution  to 
combat  outcomes  of  IEW/TA  systems  as  it  allows  us  to  represent  our  understanding  of 
the  complex  effects  and  relationships  that  characterize  warfare  and  to  observe  the 
implications  of  our  beliefs  for  different  cases  under  varied  assumptions.  Unfortunately, 
intelligence  and  electronic  warfare  are  not  well  represented  in  existing  combat 
simulations.  There  are  fundamental  technical  reasons  why  this  is  so,  and  technical 
challenges  that  must  be  addressed  to  support  the  analysis  of  the  combat  value  of  IEW/TA 
with  simulation  models.  This  document  describes  these  challenges  and  considers  how 
they  might  be  met 

Not  only  will  it  be  necessary  to  adequately  represent  the  intelligence  process  in  a 
simulation  model,  but  a  greater  and  more  fundamental  challenge  is  to  take  into  account 
potentially  large  sensitivities  of  combat  outcomes  to  multiple  assumptions  having 
unresolvable  uncertainties. 

Building  combat  simulations  requires  making  assumptions  with  considerable 
uncertainty.  For  example,  often  the  performance,  effectiveness,  and  failure  rate  of 
systems  in  real  combat  can  be  estimated  only  from  limited  test  or  historical  data. 

Initiating  scenarios  and  force  dispositions  can  only  be  guessed  at  Various  human  factors 
that  make  real  battles  different  from  technical  estimates  must  be  allowed  for  in  any 
simulation  that  is  to  be  used  to  assess  outcomes  of  actual  combat.  These  must  be  arrived 
at  through  a  combination  of  historical  analysis  and  updating  to  allow  for  modem 
weaponry. 


Uncertainties  pose  a  problem  for  die  use  of  simulation  in  assessing  combat 
outcomes  regardless  of  the  nature  of  the  simulation.  Furthermore,  the  inclusion  of 
IEW/TA  causes  a  large  increase  in  the  amount  of  uncertainty  above  what  must  be 
addressed  in  simulations  directed  primarily  toward  lethal  systems.  Consequently,  the  use 
of  simulation  in  evaluating  the  contribution  of  IEW/TA  to  combat  outcomes  requires 
innovative  approaches. 

There  are  two  reasons  why  IEW/TA  presents  greater  inherent  problems  with 
uncertainty:  soft  (human)  factors  and  nonlinearity. 

SOFT FACTORS 

The  product  of  intelligence  is  information.  Intelligence  influences  the  decisions  of 
human  actors.  Simulating  this  effect  requires  modeling  the  decision  process.  For  many 
purposes,  all  of  the  basic  command,  control,  and  communications  network  functions  that 
support  decisionmaking  must  be  modeled  as  well.  *  *odels  of  human  decisionmaking  are 
generally  inadequate  and  in  any  case  are  extremely  difficult  to  validate.  Decision  models 
must  in  general  represent  a  variety  of  "soft"  or  psychological  factors  that  are  difficult  to 
tie  to  available  data.  These  soft  factors  create  serious  problems  for  representation  and 
for  calibration  of  the  model.  Combat  models  that  focus  on  lethal  weaponry  often  neglect 
soft  factors.  This  neglect  is  sometimes  justified  either  because  human  factors  are 
believed  to  be  less  critical  than  "hard"  technical  characteristics  or  because  there  are  so 
many  human  actors  involved  that  their  actions  can  be  represented  statistically  by 
aggregate  probabilities,1  depending  upon  the  application.  The  effect  of  IEW/TA  upon 
combat  outcomes  through  target  acquisition  could  perhaps  be  modeled  by  incremental 
changes  in  probabilities  of  kill.  However,  IEW/TA’s  contribution  to  situation  assessment 
requires  that  the  decision  process  of  individual  commanders  be  explicitly  modeled. 
Because  intelligence  can  affect  critical  decisions  by  individual  decisionmakers,  no  appeal 
to  statistics  is  possible,  and  the  necessity  of  modeling  the  decision  process  cannot  be 
avoided.  Focusing  only  upon  target  acquisition  would  provide  a  more  tractable  problem 
but  would  systematically  underestimate  the  value  of  IEW/TA.  Furthermore,  situation 

1The  use  of  aggregate  parameters  ultimately  must  be  based  upon  an  appeal  to  the  "law 
of  large  numbers,"  where  the  summed  effect  of  many  human  actions  is  modeled  by  die 
average  over  the  ensemble  of  individual  acts.  This  assumption  will  not  be  strictly  valid 
In  all  cases  but  may  serve  as  a  useful  approximation  where  no  other  approach  is 
available. 
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assessment  contributes  to  decisions  and  resulting  situations  that  may  increase  or  decrease 
the  number  and  type  of  targets. 

NONUNEARIT1ES 

Intelligence  is  essentially  catalytic  in  its  effect,  and  like  other  sorts  of  catalytic 
phenomena  IEW/TA  has  a  highly  nonlinear  effect.  Intelligence  by  itself  has  no  combat 
effect,  but  it  enhances  the  effectiveness  of  other  force  elements.  In  situations  involving 
overwhelming  force  ratios,  the  combat  value  of  intelligence  could  be  negligible  and  have 
no  effect  on  the  outcome  whatsoever.2  When  a  single  command  decision  means  the 
difference  between  victory  and  defeat  (for  example  deciding  when  to  commit  strategic 
reserves),  however,  the  effect  of  oik  piece  of  critical  intelligence  is  so  nonlinear  as  to  be 
essentially  discontinuous.  Strong  nonlinearities  in  a  model  act  as  an  "uncertainty 
amplifier."  Small  changes  in  inputs  can  result  in  large  swings  in  outputs,  so  even  small 
uncertainties  in  assumptions  can  result  in  large  uncertainties  in  outcome.3  The  point  is 
that  no  single  case  may  be  "representative.”  To  evaluate  the  contribution  of  an  IEW/TA 
system,  its  effect  upon  a  wide  range  of  plausible  situations  may  need  to  be  evaluated. 
How  these  various  cases  should  be  weighted  in  a  summary  evaluation  will  often  depend 
upon  what  use  the  summary  "measurement"  is  put  to. 

These  two  aspects  of  intelligence  and  electronic  warfare  have  a  series  of  practical 
consequences  that  have  caused  most  combat  models  to  avoid  confronting  the  problems  of 
modeling  IEW/TA  head  on.  Prominent  among  these  consequences  are: 

•  Increasing  the  difficulty  of  sensitivity  analysis.  Because  the  effects  of 
intelligence  can  be  highly  nonlinear  or  discontinuous,  doing  a  thorough 
sensitivity  analysis  is  essential  if  the  outputs  of  the  model  are  to  be  used  with 
confidence.  Without  adequate  sensitivity  analysis,  study  conclusions  could  be 
misleading.  At  the  same  time,  modeling  IEW/TA  increases  the  amount  of 
uncertainty  that  must  be  addressed.  Sensitivity  analysis  may  also  become 
more  difficult  because  analyzing  the  effects  of  single  factor  sensitivities  no 

2lf  one  is  out  of  bullets,  knowing  where  all  the  potential  targets  are  to  high  accuracy  is 
of  little  help. 

3 In  the  extreme  case,  nonlinearities  can  result  in  behavior  that  is  formally  chaotic 
(Gleick,  1987).  However,  even  where  chaos  is  not  present,  nonlinearities  can  strongly 
amplify  uncertainties  for  some  fractions  of  possible  uses. 


longer  suffices.  With  strong  nonlinearities,  one  must  become  concerned  with 
multiplicative  effects  and  cross  factors. 

•  Producing  strong  scenario-dependencies.  The  operational  value  of  IE W/TA 
will  often  be  very  scenario-dependent,  and  the  details  of  the  model  needed  to 
capture  the  interesting  effects  may  also  vary  with  die  scenario. 

Consequently,  a  large  range  of  cases  may  need  to  be  run.  Producing  a  single 
model  that  will  be  valid  across  all  those  cases  may  be  quite  difficult 

•  Making  human  judgment  indispensable.  Human  expertise  is  necessary  in 
both  construction  of  models  and  monitoring  of  case  runs  to  insure  that 
assumptions  essential  for  the  validity  of  a  particular  model  are  not  violated  in 
a  scenario  under  consideration. 

•  Creating  large  issue! answer  complexity.  Because  of  the  strong  situation- 
dependence  of  outcomes,  simple  answers  to  questions  of  the  operational 
value  of  IE  W/TA  may  be  few  and  far  between.  Instead,  the  results  will  need 
to  be  carefully  bounded,  and  conditional  factors  will  need  to  be  kept  clear. 
Simulations  to  support  evaluations  of  IE  W/TA  need  to  be  able  to  manage  this 
complexity  of  results. 

These  problems  imply  that  traditional  methods  of  combat  simulation  may  be 
inadequate  to  support  the  evaluation  of  IEW/TA  systems.  The  OPVIEW  project  is 
pursuing  a  novel  approach  to  providing  computational  support  for  studies  evaluating 
IEW/TA. 

This  Note  describes  issues  that  must  be  addressed  if  the  contribution  of  IEW/TA 
systems  to  combat  outcomes  is  to  be  reliably  determined  through  the  use  of  simulation 
models,  including  the  extreme  sensitivity  of  combat  outcomes  to  inherent  uncertainties 
and  the  necessity  of  including  the  effects  of  human  factors.  Implications  are  drawn 
regarding  desirable  design  features  of  computer  simulations  to  support  studies  of 
IEW/TA’s  contribution  to  combat  outcomes.  A  proposed  modeling  environment  to 
support  the  construction  of  such  simulations  is  also  described. 
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II.  UNCERTAINTY  AND  SENSITIVITY  ANALYSIS 


When  uncertainties  cannot  be  resolved  through  empirical  research,  they  must  be 
addressed  in  the  modeling,  either  through  the  use  of  probabilistic  models  or  through 
sensitivity  analysis.  In  sensitivity  analysis,  possible  errors  in  inputs  to  the  model  are 
estimated  and  corresponding  error  estimates  on  the  outputs  are  calculated. 1  For  arbitrary 
simulations  the  oily  method  for  doing  this  is  to  run  excursions.  For  each  case,  multiple 
runs  must  be  made  to  investigate  how  the  outputs  change  as  the  inputs  are  varied  inside 
their  corresponding  ranges  of  uncertainty.  Unfortunately,  the  number  of  runs  required  to 
do  this  completely  rises  geometrically  with  the  number  of  uncertain  factors. 
Consequently,  for  most  realistic  simulation  models,  the  number  of  runs  per  case  required 
for  a  thorough  sensitivity  analysis  is  astronomical.  In  probabilistic  modeling,  values  for 
uncertain  quantities  are  selected  at  random  and  the  model  is  run  many  times  to 
characterize  the  range  of  plausible  behaviors  the  model  is  capable  of.  Probabilistic 
modeling  can  be  viewed  as  an  automatic  mechanism  for  performing  sensitivity  analysis 
and,  when  the  number  of  uncertainties  is  large,  will  also  require  a  prohibitive  number  of 
runs  to  be  efficacious. 

If  simulation  models  are  to  make  a  contribution  to  evaluating  IEW/TA,  the 
number  of  excursions  required  to  bound  output  uncertainties  must  be  kept  tractable.  One 
approach  is  to  use  aggregated  models  so  that  the  number  of  uncertain  factors  is  kept 
small.  Another  possibility  in  addition  to  the  bottom  up  approach  of  performing 
sensitivity  analysis  on  cases  is  to  utilize  a  top  down,  or  question-driven,  strategy  of 
structuring  a  series  of  cases  to  maximize  their  contribution  to  useful  answers  in  spite  of 
the  uncertainties.  This  is  done  by  building  and  using  simulation  in  order  to  best  inform 
the  answer  to  specific  questions,  rather  than  using  general  models  whose  design  is  data 
driven.  In  spite  of  the  uncertainties,  there  is  still  much  that  we  do  know,  and  there  are 
decisions  that  will  be  made  regardless  of  how  uncertain  the  world  is.  Any  analysis  of 

1  Uncertainties  in  parameters  internal  to  a  model  are  just  as  problematic  as 
uncertainties  in  quantities  that  are  formally  "inputs."  Thus  an  adequate  sensitivity 
analysis  must  consider  sensitivities  to  assumptions  made  in  building  the  models  as  well 
as  those  made  in  choosing  inputs  for  a  particular  run.  All  such  quantities  must  be 
regarded  as  "inputs"  to  the  modeling  process,  whether  they  are  represented  in  the 
computer  program  as  internal  parameters  or  as  external  inputs. 
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value  must  look  for  some  way  to  choose  rationally  among  alternatives.  For  example,  in 
choosing  between  alternative  force  structures,  considering  what  results  would  suffice  to 
indicate  a  choice  can  produce  a  list  of  cases  that  can  that  dictate  which  phenomena  must 
be  represented  and  constrain  the  process  of  sensitivity  analysis. 

Question  driven  strategies  can  be  understood  as  an  approach  to  sensitivity  analysis 
based  on  a  strategy  of  sampling  the  space  of  plausible  cases.  Where  an  exhaustive 
sensitivity  analysis  is  impossible,  it  may  still  be  possible  to  estimate  plausible  ranges  cm 
outcomes  by  sampling  the  list  of  cases  that  might  be  run  in  a  thorough  analysis.  Formal 
derivation  of  a  statistically  correct  sampling  method  will  typically  not  be  possible,  so 
selection  of  the  sample  set  will  depend  upon  human  judgment  This  expert  judgment  will 
take  the  form  of  an  analytic  strategy  in  which  critical  cases  are  used  to  establish  bounds 
on  plausible  outcomes  given  assumptions.2  For  example,  an  entire  class  of  cases  may  be 
argued  to  be  bounded  by  (be  worse,  better,  greater,  or  less  than  on  some  scale)  a  critical 
case.  In  general  this  analytic  strategy  will  not  be  initially  specified  in  full  detail.  Instead, 
it  may  often  emerge  through  a  process  of  exploration  using  preliminary  models.  The 
emphasis  will  not  be  on  finding  the  "best  estimate"  outcome,  but  on  supporting  a  line  of 
reasoning  that  holds  in  spite  of  existing  uncertainties. 

The  process  of  question  driven  modeling  implies  a  different  style  of  model 
development  than  has  often  been  the  case.  Whereas  most  modeling  efforts  begin  with 
decisions  about  data  representation  and  scope,  modeling  driven  by  the  question  must 
consider  what  hypothetical  arguments  might  suffice  to  answer  that  question  before 
deciding  upon  model  characteristics.  Details  that  turn  out  to  be  irrelevant  to  a  particular 
question  need  not  be  included. 

A  complete  sensitivity  analysis  may  be  impossible  even  with  fairly  aggregated 
models.  Thus  the  process  of  determining  the  implications  of  what  is  known  is  a  search 
through  the  universe  of  possible  outcomes  defined  by  what  is  not  known.  One  approach 
to  structuring  this  search  is  to  use  selective  resolution:  Preliminary  modeling  with  highly 
aggregated  models  is  followed  by  a  thorough  sensitivity  analysis  at  that  level  to  indicate 
where  more  detailed  modeling  could  have  the  greatest  influence  on  the  final  answer. 

This  process  can  be  done  iteratively,  with  partial  results  guiding  the  process  of  model 
development,  and  with  detail  added  only  where  it  makes  a  difference.  The  result  is  a 

^are  must  be  taken  that  the  range  of  cases  considered  are  not  selected  to  produce  the 
answer  the  analyst  would  like  to  see.  Whatever  criteria  are  used  for  case  selection 
should  be  retained  in  the  final  documentation  so  that  results  can  be  objectively  reviewed. 
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process  where  human  judgment  and  machine  computation  both  contribute  to  the  evolving 
analysis. 

For  IEW/TA  systems,  a  considerable  portion  of  the  value  derived  from  situation 
assessment  comes  horn  exploiting  special  opportunities  and  avoiding  being  surprised. 
Consequently,  the  incremental  value  of  an  IEW/TA  system  will  manifest  itself  primarily 
in  special  cases  where  such  surprises  or  opportunities  exist  To  best  estimate  the  added 
value  of  an  IEW/TA  system,  such  critical  cases  should  be  oversampled.  A  generic  study 
might  involve  discovering  "interesting"  cases  through  an  exploration  of  the  space  of 
plausible  models  and  then  studying  the  implications  of  different  IEW/TA  attributes  on 
those  cases.  The  process  could  well  be  iterative  as  new  cases  are  identified  in  the  course 
of  analysis. 

Wherever  possible,  OPVIEW  will  utilize  highly  aggregated  parameterized  models 
to  support  the  evaluation  process.  Excursions  to  compensate  for  uncertainty  regarding  a 
parameter  are  easily  generated  as  parameter  values  can  be  quickly  changed.  However, 
assumptions  that  cannot  be  adequately  parameterized  may  also  have  associated 
uncertainties.  These  nonparametric  uncertainties  affect  model  structure  and  require  the 
use  of  alternative  models  to  investigate  their  implications.  Consequently,  exploratory 
modeling  will  in  general  requite  the  use  of  models  embodying  alternative  assumptions 
(e.g.  commander  decision  style).  These  alternative  models  may  differ  from  one  another 
only  in  limited  ways.  Consequently,  their  use  does  not  require  generating  many  different 
models.  Instead,  investigation  of  the  effects  of  minor  changes  can  be  facilitated  by 
support  for  an  understandable  and  modifiable  model  code. 

One  important  source  of  uncertainty  that  is  not  parametric  lies  in  the  choice  of 
combat  scenario.  (A  given  scenario  may  of  course  incorporate  important  parametric 
uncertainties  such  as  mobilization  duration.)  To  avoid  the  time-consuming  task  of 
scenario  development  for  a  large  number  of  cases,  the  OPVIEW  environment  is  intended 
to  include  a  database  of  combat  vignettes,  which  will  provide  the  basis  for  exploring  the 
range  of  combat  outcomes  through  either  their  direct  use  or  revision  to  suit  a  particular 
study. 

In  summary,  modeling  IEW/TA  with  support  for  aggressive  sensitivity  analysis 
may  require  the  following: 
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•  Aggregated  modeling  limits  both  the  number  of  uncertainties  and  the  time  for 
individual  runs. 

•  Question  driven  model  development  provides  focused  modeling  limiting 
irrelevant  factors. 

•  Analytic  strategies  provide  a  "top  down"  structuring  of  the  list  of  cases  that 
must  be  run. 

•  Selectively  varying  the  resolution  of  the  models  allows  for  structuring  the 
search  of  the  universe  of  cases  and  optimizing  the  utilization  of  analytic 
resources. 

•  Transparent  modeling  allows  for  rapid  model  revision,  facilitating  the  use  of 
multiple  model  variants  for  exploring  the  effects  of  nonparametric 
assumptions. 
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III.  SIMULATION  ENVIRONMENTS  TO  SUPPORT  THE  EVALUATION 

OF  COMBAT  OUTCOMES 


Uncertainties  present  two  fundamental  technical  challenges:  how  to  provide  the 
capability  to  run  all  the  cases  necessary  to  take  unresolvable  uncertainties  into  account, 
and  how  to  make  the  resulting  voluminous  computer  output  produce  a  coherent  answer. 

KEY  FEATURES  OF  NEEDED  COMPUTER  SOFTWARE 

Large  simulation  modeling  efforts  have  often  encountered  difficulty  because  of 
the  complexity,  unwieldiness,  and  lack  of  focus  of  models  designed  to  serve  a  wide  range 
of  purposes.  These  problems  have  the  potential  of  being  especially  troublesome  for  the 
simulation  of  IEW/TA  effects  on  battle  outcomes.  Consequently,  OPVIEW  will  need  to 
employ  a  form  of  "exploratory"  modeling  in  which  the  analyst  starts  from  the  policy 
questions  and  works  in  a  top  down  fashion  to  call  into  play  only  those  elements  that 
prove  relevant  The  result  of  this  method  is  that  the  model(s)  used  for  each  study  will  be 
somewhat  different  built  using  the  tools  OPVIEW  provides  and  relying  upon  the  basic 
framework  already  built  into  OPVIEW’s  submodels. 

The  traditional  approach  to  the  development  of  computer  models  may  be 
described  as  a  two  phase  method.  Phase  one  is  the  construction,  verification,  and 
validation  of  a  model.  Phase  two  is  conducting  a  study  by  running  cases  on  the  model. 
Modeling  IEW/TA  in  this  fashion  would  result  in  answers  without  associated  error 
estimates,  and  consequently  would  be  ill  advised.  To  produce  a  single  valid  model  for  all 
cases  of  interest  would  prove  quite  difficult,  and  adequate  sensitivity  analysis  on  such  a 
model  would  effectively  be  impossible.  Furthermore,  the  two  phase  approach  tends  to 
disenfranchise  human  experts  who  are  not  modelers,  as  the  model  that  emerges  from 
phase  one  is  usually  so  complex  as  to  impede  man-machine  teaming  in  understanding  the 
dependencies  of  the  results  of  the  model  runs. 

We  are  now  at  a  point  in  history  where  there  are  other  possibilities  besides  this 
two-phase  approach  to  modeling.  One  of  the  motivations  for  the  two-phase  approach  has 
been  the  hope  that  valid  models  for  complex  systems  could  be  created  by  modeling  the 
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system  in  as  much  detail  as  possible.  It  is  now  recognized  that  such  a  hope  is  based  upon 
false  reductionism,  a  belief  that  more  details  will  lead  to  a  more  veridical  model.1 

The  two-phase  approach  to  modeling  reflects  an  earlier  period  in  computer 
technology  when  computers  were  expensive  and  inaccessible,  and  there  was  little  support 
for  facile  interaction  between  programs  and  users.  Computer  technology  has  now 
matured  to  a  point  where  an  interactive  approach  to  modeling  is  feasible. 

Interactive  modeling  means  that  model  construction  occurs  iteratively,  proceeding 
in  parallel  to  die  running  of  cases  and  guided  by  the  progress  of  the  analysis.  It  implies 
that  the  analysis  will  depend  not  on  a  single  monolithic  model,  but  rather  upon  an 
ensemble  of  models  (some  of  which  may  be  variants  of  a  common  root  model).  This 
style  of  analysis  requires  not  only  a  computer  model  but  an  entire  environment  to  support 
the  iterative  and  adaptive  modeling  process.  With  such  an  environment  it  will  be 
possible  to  conduct  analyses  in  a  much  more  flexible  fashion  than  has  been  possible  in 
the  past 

This  flexibility  will  require  computer  tools  that  support  facile  model  revision. 
Examples  of  candidate  tools  are  the  RAND-ABEL®  language  (Shapiro  et  al.,  1985, 
1988),  which  greatly  facilitates  ease  of  revision,  and  libraries  of  potentially  useful  model 
components.  Parameterized  baseline  models  can  support  exploration  through  parameter 
setting  and  model  revision.  The  baseline  models  could  also  allow  a  limited  range  of 
variable  resolution.  Preliminary  modeling  could  then  be  done  by  turning  off  major 
sections  of  the  models  and  setting  parameters  in  the  resulting  highly  aggregated  version. 
Such  model  variants  could  be  produced  through  simple  revisions  to  RAND-ABEL 
decision  tables. 

A  potential  problem  created  by  frequent  model  revision  is  one  of  configuration 
management  We  hope  eventually  to  introduce  tools  into  the  OPVIEW  environment  to 
mitigate  this  problem. 

The  modeling  of  IEW/TA’s  contribution  to  combat  outcomes  could  be  greatly 
facilitated  by  a  computer  environment  to  support  modeling  through: 

Unless  all  of  the  parameters  of  a  detailed  model  can  be  determined  from  more 
accurate  data  than  their  corresponding  aggregate  parameters,  the  addition  of  detail  is 
likely  to  reduce  the  accuracy  of  the  resulting  model.  The  sum  of  many  measurements 
will  generally  have  more  error  than  a  single  measurement  of  the  sum.  Where  there  is 
uncertainty,  a  detailed  model  will  contain  many  more  uncertain  assumptions  than  will  an 
aggregate  one.  Clearly,  what  is  desired  is  the  correct  amount  of  detail  for  the  model’s 
purpose.  As  this  purpose  varies,  so  must  the  level  of  detail. 
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•  A  programming  language  and  software  tools  supporting  interactive  and 
adaptive  modeling. 

•  Aggregated  and  parameterized  baseline  models  that  may  be  modified  in  the 
context  of  specific  studies. 

•  Transparent  model  computer  code. 

•  Standard  database  and  graphical  display  tools. 

•  Configuration  management  tools. 

These  facilities  will  allow  a  large  number  of  cases  to  be  run  in  support  of  any 
specific  analysis  of  combat  value.  Generating  these  runs  may  be  regarded  as  the  first  half 
of  an  analysis  of  combat  value.  The  second  half  is  to  generate  a  coherent  picture  of  all 
the  relevant  tradeoffs,  to  boil  down  the  complexities  of  the  sensitivity  analysis  to  a 
representation  that  will  provide  useful  input  to  decisionmaking. 

The  results  of  cases  that  are  run  provide  a  database  of  assumption-outcome  pairs 
that  then  present  a  problem  in  data  modeling  or  data  synopsis.  From  the  complexities  of 
multiple  cases,  useful  conclusions  are  sought  The  task  of  finding  coherent  meaning  in 
complexity  falls  primarily  on  the  insight  skilled  human  analysts  provide.  In  this  task  the 
analyst  can  be  assisted  by  computer  software  that  facilitates  manipulation  and  display  of 
the  data. 

The  process  of  exploratory  simulation,  which  could  be  used  to  produce  an  analysis 
of  the  sensitivity  of  combat  outcomes  to  IEW/TA  capabilities  in  the  context  of  other 
assumptions,  will  result  in  voluminous  amounts  of  complex  data.  This  complexity  must 
be  summarized  in  a  way  that  is  useful  to  potential  consumers  of  the  analysis,  requiring 
the  active  involvement  of  human  analysts  but  greatly  facilitated  by  appropriate  computer 
support  In  effect  the  simulation  cases  produce  a  database  of  outputs,  so  powerful  tools 
are  also  needed  to  support  the  examination  of  this  database.  Taken  together,  the  entire 
system  can  be  described  as  a  computer  tool  for  visualizing  decision  spaces  with 
uncertainty. 

SIMULATION  ENVIRONMENT  FEATURES 

The  features  described  above  constitute  a  specification  for  a  general  computer 
modeling  environment.  Such  an  environment  would  support  a  highly  interactive  style  of 
use  where  a  smart  user  (not  necessarily  a  computer  expert)  can  explore  the  implications 
of  various  situations,  intel  system  mixes,  employment  strategies,  and  so  forth. 
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An  environment  developed  by  the  OPVIEW  project  (referred  to  as  die  OPVIEW 
environment)  should  include  a  set  of  highly  aggregated,  parameterized  baseline  models 
capable  of  being  modified  during  the  course  of  the  analysis.  These  models  should  be 
constructed  in  a  language  and  style  that  allow  people  who  are  smart  about  intel  but  not 
necessarily  geniuses  with  computers  to  examine,  critique,  and  modify  them.  The  models 
produced  would  ideally  support  selectively  variable  resolution  where  preliminary 
analysis  uses  highly  aggregated  models.  Based  on  this  preliminary  analysis  a  limited 
number  of  higher  resolution  cases  may  be  specified,  where  both  the  case  specification 
and  the  parameters  that  are  disaggregated  are  suggested  to  make  a  critical  difference  to 
the  final  answer  in  the  study  of  interest.  The  higher  resolution  model  used  could  either 
be  a  revision  of  the  baseline  models  or  another  high  resolution  model  the  Army  favors. 
When  used  to  suggest  cases  for  a  higher  resolution  model,  the  OPVIEW  environment 
could  be  seen  as  a  sensitivity  analysis  tool  used  to  augment  the  utility  of  the  larger  model. 

The  modeling  process  OPVIEW  is  pursuing  depends  critically  upon  computational 
facilities  that  have  only  recently  become  available.  What  needs  to  be  constructed  is  not 
so  much  a  single  combat  model  as  a  modeling  environment,  containing  models, 
databases,  graphical  interfaces,  and  other  tools  to  help  manage  the  complexity  of  the 
problem  being  addressed. 

In  constructing  this  environment  OPVIEW  is  utilizing  the  RAND  Analytic 
Modeling  Platform  (RAMP),  a  general  purpose  simulation  environment  generated  as  part 
of  the  RAND  Strategy  Assessment  System  (RSAS)  (Davis  et  al.,  1986;  Davis  and 
Winnefeld,  1983).  The  RAMP  supports  rule-based  modeling  using  the  RAND  ABEL 
language,  combat  adjudication  using  the  SLAND  methodology  (Allen  and  Wilson,  1987, 
1988)  and  numerous  graphical  displays,  editing  tools,  and  other  features  (see  Pig.  1). 

OPVIEW  BASELINE  MODELS 

Inside  the  RAMP,  OPVIEW  is  constructing  a  suite  of  models  (see  Fig.  2), 
including  a  combat  adjudication  model,  Red  and  Blue  versions  of  sensor  models, 
intelligence  models,  and  decision  models.  These  are  not  intended  to  be  final  products 
that  will  be  sufficient  for  any  analysis  undertaken  using  the  OPVIEW  environment  They 
should  be  regarded  as  base-line  models  that  will  be  revised  in  the  process  of  most 
analyses.  Their  function  is  to  provide  a  library  of  model  components  together  with  a 
structure  to  ease  the  process  of  model  revision. 
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Fig.  2 — Operational  value  of  intelligence  model  data 


While  the  diagram  in  Fig.  2  is  Red-Blue  symmetric,  what  goes  in  the  boxes  will  be 
asymmetric,  with  Red  systems  represented  in  the  Red  Intelligence  model.  Red  decision 
styles  in  the  Red  Decision  models,  and  Red-Blue  asymmetric  forces  in  the  combat 
adjudication  model. 

The  combat  adjudication  model  contains  a  representation  of  the  battle  and 
computes  combat  outcomes.  In  OPVIEW  it  is  based  upon  the  design  of  the  SLAND 
model  (Allen  and  Wilson,  1987, 1988)  developed  for  the  RSAS.  It  is  based  upon  a  Line 
of  Communication  (LOQ  representation  and  was  designed  to  allow  for  developing  new 
theater  models  in  short  periods  of  time.  This  will  allow  the  OPVIEW  environment  to  be 
used  in  studies  for  a  variety  of  theaters  and  levels  of  combat 

The  representation  of  ground  truth  produced  by  the  combat  adjudication  model  is 
input  to  the  two  Intelligence  models,  which  produce  respectively  a  Blue  or  a  Red  view  of 
die  battlefield,  depending  upon  what  systems  are  up  and  what  their  capabilities  are.  The 
outputs  of  the  Intelligence  model  are  based  loosely  upon  the  Intelligence  Preparation  of 
the  Battlefield  (IPB)  and  are  aggregated  in  terms  of  geographic  areas  and  types  of  target. 
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The  output  of  the  Intelligence  model  report  is  input  to  a  decision  and  execution 
process  that  produces  orders  to  forces,  tasking  to  intelligence  assets,  and  possible 
deceptions  and  countermeasures  that  affect  the  intel  process  of  the  opponent  The 
decision  models  are  not  intended  to  be  artificially  intelligent  or  to  serve  as  simulations  of 
the  decisionmaking  process  of  real  decisionmakers.  The  critical  decisions  to  be  made  are 
prescribed  by  the  combat  vignette  and  will  generally  be  described  by  a  rather  sparse 
(situation-dependent)  decision  tree.  The  computer  code  describing  these  decisions  will 
need  to  be  provided  for  each  vignette.  However,  this  case-specific  code  can  be  kept  terse 
by  use  of  utilities  for  situation  assessment,  order  generation,  and  battle  management. 

Such  utilities  will  be  provided  with  the  OPVIEW  environment 

While  most  combat  models  tend  to  become  highly  detailed,  the  OPVIEW  system 
of  models  is  designed  to  be  quite  broad  but  correspondingly  shallow.  The  intent  is  to 
represent  all  factors  that  enter  into  determining  combat  value,  even  those  that  are  not 
well  understood  and  can  be  represented  only  parametrically.  Where  high  quality 
numbers  are  available,  they  should  be  used.  Where  they  are  not,  a  parametric  approach 
will  be  taken,  with  sensitivity  analysis  determining  what  effect  these  "guesses"  have  upon 
the  final  answers. 

USING  OPVIEW  TO  EVALUATE  lEW/TA’S 
CONTRIBUTION  TO  COMBAT  OUTCOMES 

The  OPVIEW  environment  will  be  used  in  studies  to  support  running  a  large 
number  of  cases  using  various  assumptions.  The  outputs  of  all  those  cases  will  be  used 
to  produce  coherent  answers  to  the  questions  the  studies  address. 

On  any  run  of  the  model,  four  kinds  of  numbers  may  be  regarded  as  candidate 
measures  of  IEW/TA  value  (see  Fig.  3).  First  there  are  technical  parameters,  such  as 
sensor  resolution  or  range,  that  are  inputs  to  our  system.  At  the  other  extreme  are  the 
actual  combat  outcomes  such  as  attrition  or  forward  line  of  own  troops  (FLOT) 
movement.  Both  sensor  resolution  and  FLOT  movement  may  for  some  cases  represent  a 
measure  of  Intelligence  quality;  the  first  reflects  capability  and  the  second  outcome. 
Intermediate  products  of  the  simulation  may  also  serve  as  candidate  measures.  The 
outputs  of  the  intel  model  provide  measures  of  how  complete  and  accurate  Blue’s  (or 


Technical  Inputs 

•  Sensor  characteristics 

•  Platform  range,  speed 

•  Vulnerability 

•  Coverage 


Intel  Products 
•  %  of  units: 

-  Detected 
-Tracked 
-Located 

-  Identified 

-  Acquired 


Combat  Outcomes 

•  Attrition 

•  Force  ratio 

•  FLOT  movement 

•  Surprise  and  its 
avoidance 


Intelligence: 

-  Timeliness 

-  Accuracy 

-  Relevance 

-  Adequacy 


Battle  Management  Decisions 

•  Decisions  made 

•  Plans  generated 

•  Ordersissued 

•  Forces  deployed 

-  Completeness 

-  Timeliness 


Fig.  3 — Ways  OPVIEW  can  be  used  to  measure  IEW/TA ’s  contributions 


Red’s)  view  was  (or  for  countermeasures  how  deceived  Red  was).  Outputs  of  the 
Decision  model(s)  can  be  used  as  measurements  of  how  good  the  decisions  were  with 
alternative  IEW/TA  systems. 

While  combat  outcomes  are  the  ultimate  measure  of  IEW/TA  quality,  the 
volatility  of  combat  outcomes  may  often  be  such  that  they  alone  may  be  inadequate  to 
reveal  patterns  among  a  feasible  number  of  cases.  Understanding  how  IEW/TA  qualities 
lead  to  combat  outcomes  is  essential  to  being  able  to  reach  reliable  conclusions  about  the 
relation  of  combat  outcomes  to  IEWA  A  characteristics. 

The  various  technical  inputs  and  intermediate  results  constitute  a  large  pool  of 
potential  measures  of  IEW/TA  quality.  Across  a  large  number  of  runs,  a  sizable 
database  of  such  results  described  in  terms  of  all  these  potential  measures  will  be 
generated.  The  central  analytic  f>sk  beyond  selecting  cases  will  be  to  examine  that 
database  to  discover  patterns  of  correlation  from  which  causality  can  be  inferred.  The 
datahaaf  will  be  used  to  determine  which  technical  or  intermediate  measures  are  most 
predictive  of  combat  outcomes. 
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In  simple  cases  the  presence  of  a  particular  system  may  produce  fixed  changes  in 
combat  outcome.  However,  results  are  generally  more  complicated  than  that,  and  the 
detailed  results  of  a  study  may  require  a  decomposition  into  classes  of  cases  and  for  each 
class  a  portrayal  of  the  relationship  between  the  key  factors  and  combat  outcomes.2  The 
observed  relationships  may  be  conveyed  in  a  variety  of  ways  depending  upon  its 
complexity,  including  the  use  of  graphs,  tables,  or  even  a  summary  spreadsheet  model  to 
capture  the  pattern  of  results  seen  in  the  multiple-case  runs.  These  various 
representations  constitute  simplified  models  synopsizing  the  general  behavior  of  more 
complex  models  inferred  from  numerous  cases.3 

To  understand  the  contribution  IEW/TA  makes,  we  must  also  determine  when  that 
effect  will  obtain  and  why  it  occurs.  Understanding  not  only  the  "what"  but  the  "when" 
and  the  "why"  is  essential  if  we  are  to  understand  the  outcomes  of  studies  and  convince 
others  of  their  veracity.  The  OPVIEW  simulation  environment  should  be  useful  for 
asking  not  only  "What  is  the  value  of  an  IEW/TA  system?"  but  also  "When  (under  what 
conditions)  will  it  have  that  value?"  and  "Why  does  it  have  value?"  This  tool  could 
prove  useful  for  examining  alternative  mixes  of  systems,  understanding  synergistic 
effects  (e.g.  cueing),  and  seeing  implications  for  employment  issues.  Consequently, 
although  OPVIEW  has  focused  on  measuring  the  combat  value  of  IEW/TA,  the  product 
should  provide  decision  support  for  a  broad  range  of  issues  as  well  as  measurements. 

Furthermore,  the  methodology  OPVIEW  is  pursuing  does  not  lead  to  a  "magic 
yardstick"  providing  hard  and  fast  measure  of  combat  value.4  Instead,  it  should  endeavor 
to  produce  a  decision  aid  that  will  illuminate  tradeoffs  and  essential  uncertainties.  Such  a 
tool  would  be  of  value  to  decisionmakers  who  must  consider  costs  and  risks  as  well  as 
benefits. 

2Depending  upon  the  study,  combat  outcomes  may  be  aggregated  in  a  variety  of  ways 
including  average  or  worst-case  outcome. 

3Such  simplified  data  models  have  been  referred  to  as  repro  models  (Goeller  et  al., 
1985). 

4Because  of  the  complexity  of  the  phenomena  involved,  such  simple  measures  (e.g. 
JSTARS  =  3.4  armored  divisions)  will  not  be  possible  except  when  caveated  by  attendant 
assumptions. 
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